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ABSTRACT 



Whether students of various ages reported different 



self -perceptions of their writing because of survey design effects was 
studied. As part of a writing performance assessment, 1,972 students in an 
urban/suburban school district in grades 3 (n=736) , 6 (n=636) , and 9 (n=600) 

completed a brief self -perception survey that contained 5 items measuring 
students' perceived competence in writing. Students were randomly assigned to 
answer questions or to respond to statements regarding their beliefs about 
themselves as writers. Both versions of the survey contains response options 
that were ordered from positive to negative for all items, and items were 
presented in the same order on both versions. A univariate analysis that 
controlled for gender showed a two-way interaction between grade level and 
survey format. Students rated themselves higher on the question than 
statement format at all grade levels, but the difference between groups 
increased with the age of the student. Older students generated more accurate 
estimates of their competence than did younger students, but overall the 
correlations between perceived competence and overall score on the writing 
assessment were low. These results indicate that, by acknowledging the 
potential effects of survey design decision on data interpretation, 
researchers will improve the validity of their inferences in developmental 
studies as well as improving the comparability of data across researchers. 
(Contains 1 figure, 2 tables, and 12 references.) (SLD) 
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Objective 

To examine whether students of various ages report different writing self-perceptions due to 
survey design effects. 

Background 

In 1976, Shavelson, Hubner, and Stanton proposed a multidimensional and hierarchical 
model of self-concept. Since that time, hundreds of survey-based studies exploring the 
nature of students’ academic self-concept have been reported in the educational psychology 
literature. These studies tend to fall into two categories, those that further investigate the 
notions of multidimensionality and hierarchy, and those that examine the relationship 
between self-perceptions and achievement. 

Marsh and his colleagues have focused attention on the structure and specificity of academic 
self-perceptions (e.g., Marsh, Byrne, & Shavelson (1988), Visopel (1995)). While Eccles, 
Dweck and their colleagues have spend the better part of two decades studying the 
relationship between academic self-perceptions and other motivation variables that impact 
achievement (e.g., Dweck & Elliott (1983), Eccles, Wigfield, Harold, & Blumenfeld (1993), 
Gottfried (1990), Pintrich (2000)). 

These researchers have made significant and practical contributions to the field because of 
their sustained efforts over many years. These studies show, for example, that academic 
self-concept becomes more differentiated overtime, and that the relationship between 
motivation and achievement is shaped by the types of feedback students receive about their 
performance. An examination of the self-concept and motivation literature as a whole reveals 
that these researchers tend to use the same survey design within their work across time, but 
there is considerable variation in survey formats employed across researchers. 

The survey research literature suggests that survey format decisions are not trivial, and that 
changing the structure of an instrument may alter its reliability and validity (e.g., Fowler, 1993, 
1995; Schuman & Presser, 1981). In other words, we may draw different conclusions 
depending on the format selected. Little attention in the achievement motivation literature is 
paid to systematic consideration of survey format decisions and acknowledgment of the 
possible effects of these decisions on results is rarely made. 

On the other hand, the survey research literature is notably lacking in studies of survey 
design effects in non-adult populations. For example, do children respond differently than 
adults to statements with agree-disagree type scales than to items posed as questions? Are 
children more likely to choose responses that are the first ones presented in a list than are 
adults? Because surveys are commonly used in research with children, adolescents, and 
adults, these are potentially fruitful areas of inquiry. 
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Presented here, is the second in a series of studies designed to answer these types of 
questions. As part of an initial study of 320 college students, Hohn (1996) found that 
participants responded more reliably to items in question format than to agree-disagree 
items. Also, question formatted items were more predictive of grades than agree-disagree 
items. The study reported here considers item format from a developmental perspective. Do 
the same effects occur with 3 rd , 6 th , and 9 th graders as with adults? 

Method 

As part of a writing performance assessment, 1972 students in an urban/suburban school 
district in grades 3 (n=736), 6 (n=636), and 9 (n=600) completed a brief writing self- 
perception survey. The survey contained 13 items, the last 5 of which measured perceived 
competence in writing. Students were randomly assigned to answer questions or respond to 
statements regarding their beliefs about themselves as writers. Both versions of the survey 
included response options that were ordered from positive to negative for all items. Items 
were presented in the same order on both versions of the survey. 

Results 

Question 1: How do students of different ages rate their writing self-perceptions given two 
versions of the survey (question format vs. agree-disagree to statements)? 

A univariate analysis of variance controlling for gender effects showed a two-way interaction 
between grade level and survey format. Students rated themselves higher on the question 
than statement format at all grade levels, but the difference between the groups increased 
with the age of the student. 

Figure 1: Average writing self-perceptions by grade and survey format 
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Question 2: Which format is more predictive of achievement? 

Table 1 below shows internal consistency reliability estimates among the five items computed 
for each grade level and survey format. Reliabilities were not especially high, but were 
similar across grade levels and formats. 

Table 1: Internal Consistency Reliability Estimates by Grade for Survey Formats 



Grade Levels 


Question 


Statement 


Grade 3 


.66 


.65 




(n=357) 


(n=349) 


Grade 6 


.67 


.60 




(n=310) 


(n=308) 


Grade 9 


.61 


.67 




(n=302) 


(n=281) 



Table 2 shows the correlation between perceived competence and an overall score on the 
writing assessment administered just before the survey was completed. Consistent with 
other findings in the developmental self-perception literature, older students generated more 
accurate estimates of their competence than did younger students. Overall the correlations 
were low, however. It is unknown why 9 th graders were so inaccurate in rating themselves 
using the statement formatted items. 

Table 2: Correlation between Writing self-perceptions and Actual Performance by Grade for 
Survey Formats 



Grade Levels 


Question 


Statement 


Grade 3 


.15 


.09 




(n=370) 


(n=359) 


Grade 6 


.36 


.23 




(n=314) 


(n=315) 


Grade 9 


.32 


.09 




(n=303) 


(n=284) 



Conclusions 

The study reported here isolates a particular feature of survey design, namely the type of 
stem (statement vs. question). Many other survey design features deserve attention in the 
developmental literature. A few of these include: stem wordings (positive, neutral, negative), 
response option order (all positive to negative, all negative to positive, some reverse keyed 
items), and item groupings within a survey (all items about the same topic grouped together 
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or items randomly ordered). By acknowledging the potential effects of survey design 
decisions on data interpretations, researchers will improve the validity of their inferences in 
developmental studies as well as improving the comparability of data across researchers. 

Self-perceptions in academic domains are studied frequently by educational researchers. 
These findings suggest that all survey formats are not equivalent in a developmental context. 
One could easily draw different conclusions about students’ ability to accurately self-assess 
depending on one’s assumptions regarding survey format. Therefore, we may be more 
prone to drawing erroneous conclusions in the theory-building process unless further study of 
these effects is undertaken. 
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